Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(playground): accumulate errors while parsing span attributes #4941

Merged

Conversation

Parker-Stafford
Copy link
Contributor

@Parker-Stafford Parker-Stafford commented Oct 9, 2024

resolves #4922

  • accumulates parsing errors while transforming a span to a playground instance and displays them in the playground
  • displays output messages in the playground if present on the span

Errors

image

Output

Screenshot 2024-10-09 at 5 09 10 PM

@dosubot dosubot bot added the size:XL This PR changes 500-999 lines, ignoring generated files. label Oct 9, 2024
import { PlaygroundSpan } from "./spanPlaygroundPageLoader";

/**
* Checks if a string is a valid chat message role
*/
export function isChatMessageRole(role: unknown): role is ChatMessageRole {
return chatMessageRoles.includes(role as ChatMessageRole);
return chatMessageRolesSchema.safeParse(role).success;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't have to use this but i wanted to add it to the shcmea for ChatMessages so figured would update here as well

* The role of a chat message with a LLM
*/
export type ChatMessageRole = (typeof chatMessageRoles)[number];
export enum ChatMessageRole {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needed these actual values in a few places so switched to enum

import { PlaygroundInstanceProps } from "./types";

interface PlaygroundOutputProps extends PlaygroundInstanceProps {}

export function PlaygroundOutputMessage({ message }: { message: ChatMessage }) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure if we want to show this card in all cases but wanted it for span parsing, seems to be a common way to display

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wouldn't export.

What do you mean you wanted it for "span parsing" ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh yeah export was a mistake

also the parsing wording was a brain fart haha, wanted it for displaying output messages parsed from the original span** is what i meant to say

Comment on lines +194 to +208
message={{
id: generateMessageId(),
content: output,
role: ChatMessageRole.ai,
}}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can update output messages to not need id, seems fine to keep for now

// expect(getChatRole("bot")).toEqual("ai");
// expect(getChatRole("system")).toEqual("system");
// expect(getChatRole("human:")).toEqual("user");
expect(getChatRole("bot")).toEqual("ai");
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops

@@ -28,6 +29,7 @@ export function PlaygroundInstance(props: PlaygroundInstanceProps) {
return (
<PanelGroup direction={isSingleInstance ? "horizontal" : "vertical"}>
<Panel defaultSize={50} order={1} css={panelContentCSS}>
<PlaygroundInstanceErrors {...props} />
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we eventually put this in a banner as a single "warning". Can always refactor

Comment on lines 26 to 46
<Flex direction="column" gap={"size-50"}>
{parsingErrors.map((error, i) => {
return (
<Alert
variant="warning"
dismissable
key={error}
onDismissClick={() => {
updateInstance({
instanceId,
patch: {
parsingErrors: parsingErrors.filter((_, j) => i !== j),
},
});
}}
>
{error}
</Alert>
);
})}
</Flex>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe one alert with a detail element might be cleaner.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah i like that, i can refactor

import { PlaygroundInstanceProps } from "./types";

interface PlaygroundOutputProps extends PlaygroundInstanceProps {}

export function PlaygroundOutputMessage({ message }: { message: ChatMessage }) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wouldn't export.

What do you mean you wanted it for "span parsing" ?

Comment on lines +54 to +66
if (isChatMessages(instance.output)) {
const messages = instance.output;

return messages.map((message, index) => {
return <PlaygroundOutputMessage key={index} message={message} />;
});
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh as in this was the original output?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, displaying the original output but then also displaying new outputs in the same way

app/src/store/playgroundStore.tsx Outdated Show resolved Hide resolved
@Parker-Stafford Parker-Stafford force-pushed the parker/4922-accumulate-errors-while-parsing branch from f918d83 to 691beb7 Compare October 11, 2024 03:02
@Parker-Stafford Parker-Stafford force-pushed the parker/4922-accumulate-errors-while-parsing branch from 319d479 to 4efc271 Compare October 11, 2024 17:38
@dosubot dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. and removed size:XL This PR changes 500-999 lines, ignoring generated files. labels Oct 11, 2024
@Parker-Stafford Parker-Stafford force-pushed the parker/4922-accumulate-errors-while-parsing branch from 4efc271 to 02e1d82 Compare October 11, 2024 17:39
@dosubot dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:XXL This PR changes 1000+ lines, ignoring generated files. labels Oct 11, 2024
@Parker-Stafford Parker-Stafford force-pushed the parker/4922-accumulate-errors-while-parsing branch from ee38d2e to 6f79a05 Compare October 11, 2024 18:28
@Parker-Stafford Parker-Stafford merged commit ead714f into playground Oct 11, 2024
7 checks passed
@Parker-Stafford Parker-Stafford deleted the parker/4922-accumulate-errors-while-parsing branch October 11, 2024 18:51
mikeldking pushed a commit that referenced this pull request Oct 11, 2024
)

* feat(playground): accumulate errors while parsing span attributes

* update tests to include message id's / tools

* update output for runs

* update comments / test names

* move parsing errors into span playground banners

* cleanup spanplaygroundpage

* add function to verify zod schema matches type
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size:XL This PR changes 500-999 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants